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Executive Summary 


California's shift towards continuous improvement in education makes 
understanding how districts and schools can learn to improve a more pressing question 
than ever. The CORE Improvement Community (CIC), a network of California school 
districts engaged in learning about improvement together, is an important testing ground 
to learn about what this work entails. 


This report continues drawing lessons from the CIC’s second year as its districts 
work together towards a common aim: to improve the mathematics achievement of 
African American and Latinx students in Grades 4-8. The CIC applies a specific continuous 
improvement approach, called improvement science, to support teams in reaching 
the aim. Improvement science, unlike many approaches to reform, is not a specific 
“program” designed to fix educators’ performance in a particular aspect of their work (e.g., 
mathematics instruction). Instead, it is an approach and tools through which educators 
can better understand the causes of lagging performance, select ideas that they believe 
will lead to improvement, test them, and collect and analyze data from those tests to 
systematically see if their ideas indeed yield better outcomes. In 2016-17, the work of the 
CIC consisted of building district teams that conducted systems analysis to understand 
the achievement gap in their respective contexts. In 2017-18, the CIC launched local 
improvement teams (LITs) at schools, which used tools and protocols of improvement 
science to identify strategies that could impact the problem of practice, test those ideas, 
and gather data about the impacts of those change ideas. 


The first section of the report briefly explains the policy context in California and 
the history of the CORE districts’ collaboration. The second part of the report details four 
major lessons learned from the CIC this year: 


1. The simultaneous goals of improving math achievement while building capacity 
for continuous improvement offer both benefits and challenges for the CORE 
Districts. 


2. Districts have a pivotal role to play in supporting and sustaining continuous 
improvement efforts focused on classroom instruction. 


5. Context matters. Preexisting structures and processes, time for educator 
collaboration, and supportive leadership all influence continuous improvement 
efforts. 


4. District and school leaders are excited about the potential of continuous 
improvement to spur deep and lasting improvement. 


The lessons learned from CORE's second year of implementing the CIC have 
broad implications throughout the state and the nation as educational organizations seek 
to bring continuous improvement to bear on problems that have been intractable to date. 
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aE] 
Introduction 


In 2016-17, the CORE hub and its eight affiliated districts launched the CORE 
Improvement Community (CIC) to support districts in accelerating the mathematics 
achievement of African American and Latinx students in Grades 4-8 while building their 
Capacity around continuous improvement more broadly.t As such, the CIC is a large-scale 
attempt to use networked improvement in education and is at the forefront of national 
interest as a result. This report describes the second year of implementation of the CIC 
by first situating this year’s work in the California policy context and an overview of 
continuous improvement methodologies. The report then homes in on the CIC, providing 
a history of the CORE Districts leading up to the initiation of the CIC and a description of 
the work of the CIC in 2016-17 and 2017-18. Finally, the report describes lessons taken 
from the experiences of the CORE Districts in 2017-18, which have broad implications for 
others interested in applying continuous improvement approaches in education. 


Setting the Stage: California's Education Policy Context 


California's public education system reaches more than six million students each 
year in nearly 11,000 schools across more than 1,000 districts (California Department of 
Education). While California public education has great strengths, it also has persistent 
challenges. For example, Getting Down to Facts Il, a set of 56 studies designed to 
document the state of California education, showed that despite recent improvements, 
California's low-income, African American, and Latinx students perform lower than others 
in the state, and that these gaps are significantly bigger than in other states (Loeb, Edley, 
mazeki, & Stipek, 2018). In recent years, as California and its districts and schools seek 
to remedy their problems, many have become interested in approaches to continuous 
improvement. Continuous improvement approaches differ from “programs” that a district 
or school might adopt to fix a problem in that continuous improvement seeks first to 
understand the places in the existing system that lead performance in a given domain 
€.g., graduation rates, suspension rates, achievement gaps) to lag behind expectations. 
Continuous improvement engages multiple stakeholders (e.g., teachers, administrators, 
operational staff, parents) in a disciplined approach to identifying, implementing, and 
testing strategies for improving performance and then spreading evidenced-based 
changes that work locally to improve outcomes (Grunow & Hough, 2018). 


In this report we use the word capacity to refer to an organization’s ability to use all of its resources to achieve its 
goals. We use the word capability to refer to an individual's skills. CORE’s professional learning programs seek to build 
participating districts’ capacity for improvement by increasing the improvement capability of their administrators and 


teachers. 
edpolicyinca.org @ 


VAPACE 


Important shifts in federal law and state accountability policy are fueling the 
interest in and spread of continuous improvement and are driving changes in how districts 
and schools think about improving outcomes for students. The 2015 reauthorization 
of national educational policy, the Every Student Succeeds Act (ESSA), uses Continuous 
improvement language. In California, increased local control through the Local Control 
Funding Formula (LCFF)* and more accountability and transparency through the Local 
Control and Accountability Plan (LCAP)° and the California School Dashboard* have 
Spurred districts across the state to focus on improving school quality and student 
outcomes. Additionally, continuous improvement has been explicitly listed as a priority by 
the State Superintendent's Accountability and Continuous Improvement Task Force® and 
the state's ESSA plan.® The Statewide System of School Support (a cornerstone of the ESSA 
plan) charges state agencies, including the County Offices of Education, the California 
Department of Education, and the California Collaborative for Educational Excellence, 
with supporting districts with struggling schools. The plan for exactly what the support 
will look like is not yet finalized; however, continuous Improvement is a central strategy. 

In fact, some would argue that, in a state as large and diverse as California, networks are 

a critical path for system improvement (Moffitt et al, 2018). In addition to policymakers, 
the philanthropic world is also looking to continuous improvement to address long- 
standing challenges (Plank, O'Day, & Cottingham, 2018). The Bill and Melinda Gates 
Foundation, for example, recently announced $92 million in grants to 19 initial grantees 
(including the CORE Districts) that will serve as hubs of improvement networks. As a 
result of the prominence, educational leaders in California see the strategic importance of 
building their organizations’ capacity to successfully implement continuous improvement 
approaches. 


However, despite the attention continuous improvement is garnering, there are 
still many questions about what it looks like in practice. Research on implementation of 
continuous improvement in education is sparse, and there is even uncertainty about what 
the term means in practice (Grunow, Hough, Park, Willis, & Krausen, 2018). There is a great 
need for educational organizations to learn by doing and share what they learn. California 
has several groups engaged in action research to apply continuous improvement theories, 
techniques, tools, and processes, including the CORE Improvement Community (CIC). 
After first providing a brief overview of continuous improvement and the history of the 
CIC, this report shares lessons for the CIC’s second year of implementation to build the 
knowledge base about what continuous improvement looks like in practice. 


2 For more information, see https://www.cde.ca.gov/fg/aa/lc/Icffoverview.asp 

3For more information, see https://www.cde.ca.gov/re/|c/ 

*To view the Dashboard, see https://www.caschooldashboard.org/#/Home 

° Their report can be found at https://www.cde.ca.gov/ta/ac/ar/account-report-2016.asp 
© See https://www.cde.ca.gov/re/es/ 
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Continuous Improvement, Improvement Science, and NICs 


There are many different approaches to continuous improvement. While they tend 
to share a focus on using data for improvement, approaches vary in terms of the roles 
practitioners vs. experts play, the way problems and solutions are identified, and the type 
of information that counts as evidence (Grunow et al., 2018). The CORE Improvement 
Community uses the approach of improvement science and draws inspiration from the 
structure of a Networked Improvement Community (NIC), as described by Bryk, Gomez, 
Grunow, and LeMahieu (2015), as a way of building a cross-organizational improvement 
network. In the report on the CIC's first year of implementation (Nayfack, Park, Hough, & 
Willis, 2017), we described the NIC approach, and its six core principles, in relative detail. 
Here we collapse those six principles about NIC function into two categories: 


1. Generating improvement: Using the tenets, tools, and processes of improvement 
science to generate ideas and approaches to accelerating student outcomes. 


2. Spreading improvement: Leveraging a networked community to accelerate the 
dissemination of learning. 


Continuous improvement as an approach to accelerating outcomes has been 
found to be effective in generating improvement in sectors as diverse as manufacturing, 
health care, and education (Liker, 2004; Rother, 2009; Kenney, 2008; Haxton & O'Day, 
2015; Aguilar, Nayfack, & Bush-Mecenas, 2017). Improvement science requires the 
engagement of frontline workers in defining problems, analyzing data to figure out how 
the current system Is producing the gap between current and desired outcomes, and 
finally generating and testing changes to address those problems. Improvement science 
also brings specific approaches and tools, such as the fishbone diagram (see a sample 
from the first year of the CIC in Nayfack et al., 2017) and the Plan, Do, Study, Act (PDSA) 
cycle, to support teams in carrying out all stages of the work. 


In addition to describing an approach to disciplined learning for the purpose of 
improvement, Bryk et al. (2015) articulated the idea that networked learning be used to 
accelerate the speed and spread of improvement. Under this approach, many teams are 
focused tightly on a well-specified aim and can rapidly test changes and share findings 
so that all teams benefit from each other's learning. As we will describe below, the 
improvement science portions of Bryk and colleagues’ theory about how a NIC works 
align well with the work of the CIC. As a network, however, the CIC diverges from the 
theoretical “networked community” in ways that are likely similar to—and therefore 
instructive for—many others in California who are influenced by the NIC model. 
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History of the CORE Districts 


In 2010, the districts that now make up CORE joined forces to write California's 
Race to the Top application. Although the application was not granted, the partnership 
sparked desire for further collaboration. The districts formalized their affiliation, and 
CORE Districts became a nonprofit organization that serves as the “hub” for the network 
of participating districts, with a governing board comprising the superintendents of the 
participating districts. The group's early efforts included collaborating to design and create 
more than 60 assessment modules to support teachers’ shift to CCSS-aligned instruction 
(Knudson & Garibaldi, 2015). In 2013 the CORE districts’ applied for and received a waiver 
from the federal No Child Left Behind (NCLB) Act to use multiple measures of school 
quality to hold schools accountable for improvement, including student social-emotional 
learning and school climate/culture survey data, in addition to the typical measures of test 
scores, high school graduation rates, and absenteeism (Marsh et al., 2016). 


After ESSA replaced NCLB in 2015 and there was no longer a need for the waiver, 
the districts reaffirmed their desire to learn from each other, share problems of practice, 
and address issues of equity. Through searching for a framework that would support 
and enhance the group's ability to leverage experiences and learning, CORE decided the 
Networked Improvement Community (NIC) model fit the organization’s needs and current 
structure: The districts were well versed in working together on meaningful problems, and 
the CORE Districts organization was already well placed to serve as a Coordinating hub. 
The districts called the newly formed network the CORE Improvement Community, or 
CIC® (Knudson & Garibaldi, 2015; Nayfack et al., 2017). 


Overview of CIC Work for 2016-17 and 2017-18 


After electing to adopt a NIC structure, CORE held the continued engagement 
of all eight districts as a top priority in working to define the specific purpose of this 
community. The CIC came together to decide on a common problem of practice to 
collectively address. Suggested areas of focus included math achievement for African 
American and Latinx students, English language arts proficiency for English Learners, and 
8th graders’ high school readiness. The districts then looked at data from across CORE 
and from each district individually surrounding these issues. Eventually, the districts agreed 
that accelerating the mathematics achievement of African American and Latinx students 
was an urgent shared problem and decided to focus on closing the gap in Grades 4-8. 


7 Only six of the current eight CORE districts (FUSD, LBUSD, LAUSD, OUSD, SFUSD, and SAUSD) were a part of the NCLB 
waiver. 

8 The CIC is one of several initiatives coordinated through CORE Districts. To learn more about other CORE initiatives 
beyond the CIC, see https://coredistricts.org/our-work/ 
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The math achievement gap was a compelling problem because it was both 
a stubborn challenge for many districts, and there were already existing data and 
measurement infrastructure around middle-level math (Nayfack at al., 2017). However, 
it's important to note that the process of deciding to become a NIC and subsequently 
searching for a common problem of practice runs counter to how Bryk envisioned a NIC 
working. Already in the problem formation process, there was compromise: Districts held 
the explicit aim about the mathematics achievement gap at varying levels of importance, 
with some primarily invested in the achievement gap as a pressing issue and others more 
interested in participating in the CIC for the sake of building capacity in improvement 
science. In Lesson 1, this report explores the tension caused by the dual nature of districts’ 
purposes in joining and staying in the CIC that stem from its very creation. 


After agreeing on a common problem of practice, the CIC conducted extensive 
systems analysis to better understand how the achievement gap looked and why it 
existed during the 2016-17 school year. To gain multiple perspectives on the system, each 
district assembled a district improvement team (DIT), consisting of leaders from different 
departments or offices, as well as at least one cabinet-level leader. DITs used continuous 
improvement tools and processes to analyze data, including standardized assessment 
scores, district assessments, student growth data, and qualitative data from interviews and 
observations they conducted. The hub supported districts in this analysis by holding formal 
convenings and training sessions, providing tools and resources, and coaching. More 
detail about the lessons learned from the systems analysis process can be found in PACE’s 
2017 report, Building Systems Knowledge for Continuous Improvement: Early Lessons 
From the CORE Districts (Nayfack et al., 2017).° 


The intensive work on systems analysis revealed that many possible solutions to the 
achievement gap needed to happen in the classroom. Accordingly, in 2017-18 the CIC 
shifted focus to building school-level teams with capacity for continuous improvement. 

In explaining the transition from a district-level focus to a school-level focus, one hub 
member said it concisely: “If you're going to improve math, you need to go where math 
happens, and where that happens is the classroom.” 


To that end, the CIC worked with districts to launch a small number of local 
improvement teams (LITs) in 2017-18. LITs were to be “test kitchens” to (a) drive progress 
towards the CIC aim by using improvement science to test changes on a small scale 
(a few classrooms in a few schools in each district), (b) decide if the changes led to 
improvement, and (c) share learning from those testing cycles. Teams typically consisted 
of teachers, school leaders, content coaches, and sometimes members of the DIT (see 
Figure 1). 


° Available at https://edpolicyinca.org/sites/default/files/building%20system%20knowledge.pdf 
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Figure 1. Nested Structures in the CIC 
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Local Improvement Team 
(School Team) 


¢ Drives progress toward the CIC aim using 
improvement science to test changes on 
a small scale 


¢ Serves as essential “learn-by-doing" space 
School Teams always include 
teachers and could also include: 

¢ School leaders 

« Content coaches 


« Members of the District Team 


This report uses data from PACE's developmental evaluation (see Figure 2) to delve 
more deeply into the experiences of these district and school team members, along with 
CORE and district leaders, as they participated in the CIC during 2017-18. 
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Figure 2. Research Methodology for Developmental Evaluation of the CIC, 2017-18 


For the 2017-18 evaluation, PACE focused on the following research questions: 


1. Can the Networked Improvement Community model, as implemented by the CORE Improvement 
Community, positively influence districts’ progress towards becoming continuous improvement 
organizations? 


a. How does the network hub influence progress? 


b. What is the process through which school districts engage in systems transformation? What are the 
policies, systems, and structures that influence this process? 


c. What do actors in different parts of the system do to positively influence the adoption, 
implementation, and spread of continuous improvement practices? 


d. What skills and supports are necessary to effectively implement tests of change locally? 
2. What are the early outcomes from the CORE Improvement Community? 


a. To what extent do case study districts show evidence of becoming more sophisticated continuous 
improvement organizations? 


b. To what extent are the CORE districts making progress towards their common aim of closing 
the achievement gap in math (Grades 4-8) for African American and Latinx students in schools/ 
classrooms where staff are engaged in improvement science activities? 


Because the CIC is only in its second year of implementation, the evaluation focused heavily on 
implementation of the CIC (Research question 1) while noting anecdotal evidence of early indicators of 
outcomes (Research question 2). Data collection activities included: 


1. Multi-day site visits by research teams to three districts, which included interviews with teachers (n = 
10), instructional coaches (n = 4), principals (n = 7), and district-level administrators (n = 31) using semi- 
structured interview protocols; 


2. Interviews in winter and spring with district leaders in all eight districts (n = 17) and CORE staff (n = 13) 
using semi-structured interview protocols; 


5. Observations of CIC events by researchers taking detailed field notes (25 hours); and 


4. Collection of documents (e.g., convening agendas, presentation slides, photographs of team share-outs 
of thinking and local work, exit slips). 


Interviews were recorded and transcribed for analysis. The research team conducting each site visit reviewed 
transcripts, documents, observation notes, and preliminary notes made in early stages of iterative analyses and 
used these to respond to prompts in an analysis guide aligned to the research questions. The analysis guide 
supported systematic “scooping’ of relevant data (Neumann & Pallas, 2015) into a detailed profile for each 

site, which was in turn reviewed by other members of the research team to strengthen the trustworthiness of 
findings (Spall, 1998). The full research team read all three site visit analysis guides and then met for a two-day 
retreat to jointly engage in cross-case comparisons, looking for themes, similarities, and differences among the 
cases. Researchers then tested emergent themes across the broader set of data. This iterative, multiple case 
study approach is useful in uncovering information about CIC implementation and how the participants in 
various organizations and filling differing roles experienced the CIC during 2017-18. 


There are, however, some limitations to our research design. Qualitative data are useful for evaluating complex 
phenomena and understanding individuals’ perspectives on their experiences; however, they are not well suited 
to identifying causation. Similarly, the findings do not generalize to other districts or contexts; an appropriate 
use of the findings is to use them to frame hypotheses to be tested elsewhere. Finally, this study was 
conducted in the spirit of the CORE-PACE developmental evaluation partnership and should be interpreted 

as such: a report to inform CORE, the CORE districts and policymakers in the spirit of development and 
collaborative improvement. This report does not seek to provide comprehensive answers to all of the research 
questions, but rather to elevate and discuss in some detail findings that the research team believes are most 
germane to the CORE districts and others using similar methods to support system improvement. 
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Professional Supports Offered by CORE 


To build capacity in DITs and the newly launched LITs, the CORE hub carried out 
two main branches of activities this year. The first was to offer three types of learning 
events tailored for different audiences in the CIC. The second was to offer individualized 
coaching to DIT leaders and some leaders of the new LITs. 


The learning events offered in 2017-18 were: 


© Engaging 


CIC Convenings: This series of three in-person all-day meetings brought 
together district leaders and DIT members from all participating districts. 
Objectives and content for each meeting progressed through the year to match 
where teams were in their development. Thus, earlier meetings strove to help 
members understand their own roles in moving towards the goals of the CIC, 
learn how to use data, and support LITs, while the last meeting focused on 
consolidating learning, summarizing and sharing Key learnings, and reflecting 
on and celebrating the year’s progress. CORE also designed the CIC convenings 
to connect members to the purpose and narrative of the CIC and to build or 
strengthen relationships across districts. Most attendees were DIT members, 
although some LIT members attended as well. To maintain continuity between 
the in-person sessions, CORE also hosted monthly virtual meetings for DIT 
leads and data leads from each district. 


LIFT (Local Improvement Facilitators Training): This series of workshops 
supported LIT leads in acquiring the mindsets and skills needed to successfully 
facilitate school-level improvement teams. The hub conducted four two-day, 
in-person sessions, and four 60—90-minute virtual sessions between the in- 
person meetings. Outside of the sessions, participants also worked on their own 
and with their teams, and received coaching as needed. While this session was 
initially designed to build skills of people facilitating improvement teams, some 
districts successfully advocated for bringing their entire LITs so all members 
could benefit from the training in improvement science. 


ELIE (Executive Leadership and Improvement in Equity): This series of 
workshops was designed to help district leaders explore the relationship 
between improvement science and equity, and consider their own roles in 
becoming catalysts for improvement. This series focused on a broad array 

of mindsets and leadership skills foundational for leaders seeking to create 
conditions that enable continuous improvement approaches to achieve equity 
goals. There were four in-person sessions and three virtual sessions over the 
school year. Participants included superintendents, other members of districts’ 
executive cabinets, and DIT leads. 
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In addition to the structured trainings, the four hub coaches engaged with district 
leadership, DIT leads, and members to build capacity in improvement science and guide 
the work of the teams. Coaching varied from district to district, partly because the needs 
and contexts of DITs and LITs varied, and partly because the coaches brought different 
knowledge and experiences to their role. However, in general, coaching for DITs consisted 
of amember of the hub: 


¢ Providing weekly or monthly check-ins to serve as thought partners for DIT leads, 
¢ Co-planning DIT meeting agendas with DIT leads, 

e Sharing improvement science or general management tools, and 

e« Attending meetings and giving feedback on meeting facilitation. 


Ina few cases, hub members directly coached LIT leads. Additionally, the hub 
provided other kinds of training and support around continuous improvement for districts 
upon request (e.g., workshops on continuous improvement for principals). 


Launching Local Improvement Teams (LITs) 


CORE directed districts to launch LITs as places where teams in each school would 
independently test change ideas and gather evidence to see if those changes improved 
mathematics outcomes for African American and Latinx students in Grades 4-8. If there 
were improvements, the CIC network could be leveraged to accelerate, disseminate, and 
scale the improvements within and across districts. CORE did not set any guidelines for 
which schools were good candidates for LITs, and districts selected them based on their 
own criteria. The variation in selection criteria is one of the most important reasons to avoid 
making any causal inferences based on student outcomes data from 2017-18. Some districts 
selected schools because of strong and positive preexisting improvement trajectories that 
the districts wanted to learn more about through the CIC; districts selected other schools 
because they were struggling, or for a reason unrelated to student achievement. Finally, LITs 
typically did not involve all of the tested grades in participating schools. While it would be 
inappropriate to make any causal inferences about the relationship between participating 
in the CIC and student mathematics outcomes, students’ achievement status and growth 
nonetheless form an important context for the work. 


As Figure 3 shows, participating schools were as varied in terms of student outcomes 
as the CORE districts themselves. The x-axis of Exhibit 3 shows the percentage of students 
in each school meeting mathematics standards on the Smarter Balanced Assessment 
Consortium (SBAC) test. The y-axis shows a growth metric developed by the CORE Districts, 
in collaboration with Education Analytics, to estimate how much more or less students in 
a given school are learning relative to their peers in other schools. The model calculates a 
growth percentile for each student relative to other peers with similar initial achievement 
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and demographic characteristics (e.g., English learner, free/reduced price lunch). Individual 
students’ percentiles are then aggregated into an overall growth percentile for each school.” 
The red dots indicate schools with LITs; the other dots indicate the other elementary and 
middle schools in the CORE districts. Schools in the bottom left quadrant have low student 
achievement and they are also low in terms of how quickly their students are growing in 
mathematics relative to schools serving similar students. There are LITs in the bottom left 
quadrant, but there are also LITs in the upper left quadrant. In these schools, even though a 
majority of students are not yet achieving proficiency in mathematics, students are exhibiting 
above-average growth relative to other schools serving similar students. 


Figure 3. Variation in 2017-18 Mathematics Achievement and Growth for Schools with LITs 
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CORE believed that LITs should be started in places with interest and excitement 
around using improvement science to address current inequities in mathematics 
achievement, because continuous improvement is more likely to succeed when 
participants have buy-in." Down the line, CORE envisions that interest in using continuous 
improvement would spread driven by the evidence of success from the early adopters. 


10 For more information about CORE's growth metric, please see: https://coredistricts.org/our-data-research/ 
improvement-measures/ 

" The idea that top-down mandates can be problematic when work requires deep engagement is not unique to CORE. In 
fact, it has some empirical support in the broader literature on teacher learning (Kennedy, 2016). It is possible that buy- 
in is even more important in continuous improvement than other interventions because of the risk-taking required as 
frontline workers publicly examine the flaws in the current system of which they are an integral part. 
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How LITs were launched varied across districts but typically included identifying 
appropriate schools to participate, selecting members for the LIT, onboarding members 
with the purpose and processes of the LIT, and continuing to support the LITs throughout 
the school year. Figure 4 provides more detail about the big take-aways the hub learned 
from launching LITs this year. 


Figure 4. Launching LITs 


Many members of the hub reported that the learning around launching and 
supporting LITs was one of the organization's big successes of the year. While it is clear 
that to affect instructional issues using continuous improvement those delivering and 
supervising instruction (i.e., teachers and principals) need to be involved, there is woefully 
little information in the literature about how to get these practices started in schools. Based 
on their experiences supporting the launches of LITs in 2017-18, the CORE team learned 
lessons about how to get LITs started in the work. Their key takeaways from 2017-18 were: 


¢ Ahalf- or full-day team launch is helpful to set the conditions and build the baseline 
knowledge of improvement science necessary for success. All team members need 
a foundational understanding of improvement science. A short presentation can help 
build the requisite baseline Knowledge. 


e Since continuous improvement requires a culture shift, it is important to launch LITs 
with a discussion of norms and expectations for the group so that team members 
can co-construct the safe space necessary to share their instructional practices, 
reveal failures so they become opportunities for learning, and function as a 
collaborative group. 


Organizational leaders, such as school principals, may be accustomed to leading in 

a Style of the person who Is “in charge” and who “has the answers.” Those styles can 
lead one person to dominate the airtime in a group in a way that is incompatible with 
the culture necessary for highly functional LITs. 


¢ Relatedly, setting clear roles for team members can help the team function. CORE 
found that having a good team lead (based at the school site and responsible for 
scheduling and communication), improvement facilitator (a more expert peer who 
has slightly more knowledge of improvement and who can design meetings, help 
the teams select improvement tools, and generate data displays), and a project 
sponsor (a school or district leader who is generally informed about the team’s needs 
and project but is not involved in the day-to-day of enacting and testing changes) 
were all important for success. 


¢ Not all data and ways of presenting data are equally helpful. LITs often need specific 
support in thinking about the types of data they could best use to support their 
learning as well as help getting those data in usable form. One pitfall that came 
up in multiple situations was the provision of a high volume of annual school-level 
performance data. Such presentations can be overwhelming and might not be well 
matched to a specific LIT’s data needs at any given point in time. Instead granular 
and timely data related to the changes being tested gave teams actionable feedback 
necessary to learn about their tests and adjust course If necessary. 
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Most LITs began to operate in late fall or early winter. District Improvement Teams, 
with the help of CORE's professional learning opportunities (e.g., CIC, LIFT, coaching 
for DIT leads), facilitated LITs’ efforts to conduct a school-level systems analysis (e.g., by 
using empathy interviews to understand students’ experience of mathematics), choose 
a problem of practice aligned with the larger CIC aim, develop change ideas that could 
affect that problem, run PDSAs to test the change ideas, analyze the data from those 
cycles, and decide on next steps. As described in more detail in Lesson 2 below, DITs took 
on varied roles in terms of their own work and the ways they supported their LITs. Lesson 
35 explores how LITs took shape differently in their varied contexts. 


Two districts decided not to launch LITs during the 2017-18 school year. One of 
those districts already had extensive improvement structures and processes in place, 
which they believed would enable them to act on what they learned through their root 
cause analysis at a broad scale throughout the district. The other district's systems analysis 
had led them to identify a major root cause of the achievement gap—staff turnover ata 
set of schools that serves a high-proportion of the district's African American and Latinx 
students—that did not lend itself to classroom-level solutions. The DIT decided to tackle 
that issue from the district level. Both districts used continuous improvement processes to 
work towards the network's aim. 


District Implementation Progress 


With the help of the professional supports offered by CORE, all eight CORE 
districts used continuous improvement approaches to work towards the network aim of 
accelerating mathematics achievement of African American and Latinx students in Grades 
4-8. Table 1 shows the districts, how many LITs they launched, and the focus areas for 
some of the PDSAs conducted this year. While not comprehensive, this table gives a taste 
of the drivers of inequitable mathematics achievement identified by LITs. 


10 For more information about CORE’s growth metric, please see: https://coredistricts.org/our-data-research/ 
improvement-measures/ 

" The idea that top-down mandates can be problematic when work requires deep engagement is not unique to CORE. In 
fact, it has some empirical support in the broader literature on teacher learning (Kennedy, 2016). It is possible that buy- 
in is even more important in continuous improvement than other interventions because of the risk-taking required as 
frontline workers publicly examine the flaws in the current system of which they are an integral part. 
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Table 1. Illustrations of Districts’ Work in the CIC in 2017-18 


Number 


District oe) NES Focus of some of the PDSAs this year 
LETT atol at=1e | 


District A 4 ¢ Identify and incorporate representations of mathematical concepts to support 
student understanding 
¢ Use of a 10-minute “opener” with student-to-student discourse around rich tasks 


District B 7 ¢ Student reflections to identify confusion 
¢ Student questions during math class 


District C S * Test changes to improve implementation of a specific math strategy/approach, 
focusing on student conversation about math 


District D None ¢ Provide professional development on an instruction continuum that deliberately 
balances caring relationships with high expectations and supports for student 
success, with a focused effort on improving engagement and outcomes for 
African American students. 


District E 2 ¢ Improve students’ ability to decipher math problems and improve perseverance 
¢ Increase culturally responsive pedagogy 


District F 3 ¢ Increasing student dialogue about mathematics 


District G None ¢ Improve teacher recruitment and retention at schools serving a high concentration 
of African American and Latinx students 


UO 
wn 
= 
az 
ine) 
e 


mprove student collaboration in small group work in mathematics 


The eight districts created various local change ideas to help achieve the overall 
improvement goal, with distinct efforts based on local circumstances and priorities, such 
as one district's emphasis on “opening” math classes with richer mathematics tasks and 
another district's piloting of a program to build teachers’ awareness of and comfort with 
the community their schools served. Collectively, the LITs and DITs worked on testing 
change ideas to address six of the seven secondary drivers identified in the CIC's driver 
diagram for 2017-18 including: (1) Student Academic Discourse: Justifying and Critiquing 
Reasoning; (2) Students’ Use of Problem Solving Strategies; (3) Lesson Design, Delivery, 
and Use of Instructional Strategies; (4) Effective Student Group Work; (5) Attitudes, 
Behaviors, and Beliefs About Math; and (6) Professional Communities and Capacity. 
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Lessons Learned 


Because the CIC is working to support the implementation of continuous 
improvement on a very large scale, the experiences of the CORE hub, participating districts, 
and their DITs and LITs in 2017-18 offer lessons that likely have broad implications in 
California and beyond. For example, the California Collaborative for Educational Excellence 
reports support 56 professional learning networks (with a total of 300 participating LEAs) 
at the time of this writing.* Below we discuss each of the lessons, drawing evidence from 
observations of CORE events and DIT team meetings, interviews with CORE hub leaders 
and DIT leaders, and in-depth site visits to three of the CORE districts. 


Lesson 1: The simultaneous goals of improving math achievement while building 
capacity for continuous improvement offer both benefits and challenges for the CORE 
Districts. 


A key feature of the networked improvement community model is that all groups or 
organizations work on the same aim, have a shared theory of improvement, are disciplined 
in how they use data to learn how to improve, and collectively learn from each other's 
tests through a collaborative network (Bryk et al., 2015). This creates opportunities for each 
team to work deliberately even while facilitating rapid learning in the network as a whole. 
As described above, however, the CORE districts network existed several years before 
they selected an aim. Additionally, the aim of accelerating mathematics achievement for 
African American and Latinx students in Grades 4-8 through CIC work, while agreed upon 
as a network focus, is more or less central in each district's overall strategy. Finally, the root 
cause analyses conducted during the 2016-17 year led to different district-level foci, which 
fragmented further when LITs launched their own root cause analysis during the 2017-18 
school year. As a result, one of the fundamental conditions for spreading learning within an 
improvement science network—a tight focus on a shared aim—is inconsistent among the 
CORE districts. Based on the broader interest in California in building improvement capacity, 
we think that this reality is likely to be the norm rather than the exception among networks 
launched in the state. And even if networks did have a tight focus on a shared aim, it would 
be natural for participating organizations to seek to build improvement capability more 
broadly (Barletta et al., 2018). As a result, how the CIC works in an environment where there 
is not a uniformly tight focus on the stated aim could be broadly informative. 


For the CIC, the problem of accelerating mathematics achievement for African 
American and Latinx students in Grades 4—8 offers both the opportunity for improving a 
key outcome as well as an opportunity to learn continuous improvement by engaging in an 
authentic improvement task. As a CORE leader explained, “[M]uch of the time, we engage in 


2 To learn more about CCEE professional learning networks see: https://ccee-ca.org/professional-learning-networks.asp 
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work and recognize that... at the detail level there's a topic [mathematics] that we're focusing 
on, but at a meta level, we're also learning about what we're doing...” District comments 
also show a clear interest in the opportunity to learn broadly applicable continuous 
improvement approaches through investigating mathematics achievement. When describing 
the rationale for her district's engagement in the CIC, one district leader explained that “a 
deeper understanding of improvement science and how that can apply to other things 
would be an overarching thing we would like to get out of it..." The sentiment that working 
on the mathematics aim could also be a practical way to learn the skills to catalyze broader 
improvement was common for both CORE staff and district leaders. 


From both the CORE and districts’ perspectives, then, we can say that in 2017-18, the 
CIC worked towards the stated aim of accelerating mathematics achievement for African 
American and Latinx students in Grades 4-8 while simultaneous working towards a broader 
goal around helping districts improve their capacity to be a continuous improvement 
organization. While there is obvious logic to the idea of building people's improvement 
capabilities more broadly while focusing on a specific problem of practice, experiences of 
the CORE Districts in 2017-18 also snowed some places where the aim and the goal were in 
tension. The district leader quoted above laid out this tension as she continued: 


..We'd also like to get out of it-which we don't yet get out of it yet, but I’m 
hopeful—is really getting good ideas around the math thing, or expertise 
from other districts that are having promising practices. | don't think we've 
really had that happen this year because not everybody is in the same place. 


In 2017-18, there were two ways that not everybody was “in the same place’ in the 
CIC: (1) There was too much variation in capacity to do continuous improvement without 
outside assistance, and (2) the drivers DITs and LITs focused on were too varied for cross- 
district collaboration to produce the type of knowledge-sharing desired for networked 
improvement. CORE responded to the first issue by orienting CIC programming to build 
improvement capability of participating individuals and organizations. The second issue 
arises from the fact that districts remain committed to the CIC even if the work their teams 
are undertaking does not align to a tightly specified aim. The result is a structural tension 
within the CIC, where districts agreed to a broadly defined (instead of tightly specified) aim 
and so the mechanism by which networked improvement would accelerate knowledge 
Sharing in theory did not function smoothly in practice. 


One place the tension showed up was in the CIC convenings, which were a space 
for shared learning about improvement and cross-district collaboration. CORE designed 
the February 2018 CIC convening specifically to enable the DITs and LITs from across 
the eight districts to learn about each other's improvement work, with each DIT and LIT 
sharing structured storyboards about the focus of their work, successes, and challenges to 
date. The foci the teams presented on included (among many): 
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« Using “three reads,” a strategy of having students do a close reading of complex 
mathematics word problems multiple times with a different purpose each time, 
to build students’ comprehension of the tasks; 


¢« Using a different type of “opener” to increase student-to-student discourse and 
quickly transition into richer mathematics content; 


e Increasing the frequency with which students ask questions if they are confused 
in mathematics classrooms; 


e Increasing the amount of student discussion of mathematics reasoning and 
student comfort in discussing mathematics; and 
¢ Increasing teacher retention in a specific set of schools. 


As this sample shows, there was some convergence in foci around the general 
topic of student discourse in mathematics—another cluster occurred around mathematical 
models and representations—but there were also some notable outliers. As a result, 
participants had divergent experiences with the CIC convenings as a forum for leveraging 
network learning about math. Feedback exit slips from this CIC convening varied, 
including from one respondent who noted that the convening provided a “great beginning 
of sharing work across the network,” and another who appreciated the “experience of the 
showcases to get better understanding of the process.” Other participants commented 
on the “need to bring in more math content and equity” and that “the two [presentations 
from other districts] | saw had differing ideas that did not help my improvement.” Our 
analysis suggests that the work of constructing opportunities for sharing that provide value 
for all participants (while always a difficult task) is even more challenging in networks that 
are committed to building improvement capability but have not truly united around a 
tightly specified aim. One district leader reported, “I’m super interested in this idea of ‘Are 
there change ideas out there for math that people could benefit from and get more rapid 
improvement cycles going?” but focusing convenings narrowly on testing change ideas 
in a shared space would necessarily limit the participation of several districts in the CIC. In 
2017-18, CORE worked with this duality by providing opportunities for participants to learn 
improvement science through CORE events and then supporting participants to apply 
their learnings to the ways their local teams had constructed their mathematics aims. 


Lesson 2: Districts have a pivotal role to play in supporting and sustaining continuous 
improvement efforts focused on classroom instruction. 


Many attempts at reform designed to improve student achievement have preceded 
the recent attempts at bringing continuous improvement to schools. In fact, their 
persistent failure to make changes to the instructional core is part of what motivates 
people to try continuous improvement (Tyack & Cuban, 1995). We start by foregrounding 
the fact that one way improvement science seeks to break the cycles of failed reform is by 
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making the engagement of frontline workers—in the case of education these are typically 
teachers and other school-based staff—a central strategy in its approach to generating 
improvement. Not only does improvement science seek to engage them in order to 
create motivation and buy-in but also it recognizes that the people doing the work are 
uniquely positioned to understand the problems in the current system and to generate 
and test improvements. The engagement of the frontline, however, does not imply an 
abdication of responsibility and leadership from the top. Past reform efforts have shown 
that not only top-down but also bottom-up reforms are likely to fail-though for different 
reasons (Coburn, 2003). To achieve its potential, continuous improvement appears to 
require a coordinated and multilevel approach to reform. 


As described above, the CIC moved from a focus on the district level in 2016-17 to 
a focus on local improvement in 2017-18 because inequitable mathematics achievement 
has an instructional component that cannot be addressed solely from the district level. 
This does not imply that the district role is less important than previously thought, just that 
districts need to play more roles when improvement efforts have school-level dimensions. 
With that in mind, we describe the roles districts played in ClC-related continuous 
improvement work in 2017-18. Important roles included: 


1. Being a champion; 
2. Ensuring sufficient resources for LITs to support improvement efforts; 


3. Coordinating up, down, and across the system (e.g., classroom, school, and 
district level); and 


4. Conducting their own district-level improvement cycles. 
We describe what each of these looked like In the CIC in 2017-18 below. 


Being a champion for an initiative is a critical role, and it may be especially true 
for continuous improvement because it requires both resources and culture shifts, such 
as Sharing failures publicly so that they can be opportunities for collective learning. One 
district leader explained this part of the role by saying, “| very much see myself as an 
encourager, a connector, as a coordinator in terms of bringing people together when 
that needs to happen, consolidating what we're learning, facilitating reflection and then 
facilitating strategy moving forward.” Champions may also be in critical positions to model 
the culture shifts by leading in ways that model distributed leadership, rewarding people 
for innovating and learning from mistakes, and taking the time to diagnose problems 
before selecting solutions. One final role for champions is to use their positional authority 
to clear away other competing responsibilities and initiatives so that teams can focus 
on continuous improvement. For example, one of the CORE districts had an existing 
professional learning structure where elementary school teachers were supposed to 
undertake multiple cycles of improvement over the year, distributing those cycles across 
the core subjects. Teachers in one LIT requested an exemption from the planned subject 
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rotation so that they could have a consistent focus on the aim that was the heart of their 
CIC work. The champion was able to grant that waiver, facilitating deeper engagement in 
hie Gic. 


Ensuring sufficient resources to support improvement efforts came up as a 
crucial role for districts in supporting continuous improvement. The most common 
challenge to doing continuous improvement work cited by DIT leads and LIT members 
was finding a regular and uninterrupted time to meet. This seems to be especially true 
for teachers, whose roles as currently conceived provided relatively little time for parts 
of their job other than instructing students. Districts in the CIC found ways to address 
the issue of teacher time. In one school, where the collective bargaining agreement 
limited administrators’ ability to control teacher collaborative time within the contract 
day, a district leader was able to identify resources from a mathematics grant that could 
be used to pay for teachers’ and a math coach's time after school so that the LIT could 
meet. In other cases, LITs used teacher collaborative time already built into schedules 
with district or school permission to turn that time over to LIT meetings. Finally, districts 
covered teachers’ classes and reallocated district staff responsibilities to enable a range of 
personnel to attend CIC events such as the convenings, LIFT, and ELIE. 


Districts also provided resources in the form of coaches or district staff to support 
LIT efforts. A district leader explained the decision to provide math coaches to help 
support LITs: 


[Wle needed that spark, so that the teams could develop their own identity, 
and... we needed to provide the capacity to get that work started. And, we 
had talented members of our district improvement team that we thought 
could do that work..... In our case, our school teams really needed that 
coach. That district coach was huge. | think that was the primary piece. 


Many of the LITs focused on aspects of math instruction that relate to some of 
the most challenging shifts required by Common Core State Standards, and pedagogical 
approaches or curricula that are aligned to them. Teaching more conceptual mathematics 
and increasing student skills at communicating about their mathematical thinking require a 
deep and flexible understanding of mathematics content. As a result, it is unsurprising that 
many districts identified placing mathematics coaches with LITs as an important strategy 
for facilitating their success. 


Districts also dedicated data support resources to LITs. These included supports 
for a mix of quantitative and qualitative data. Although schools collect and analyze 
increasingly complex data these days, the data collected are not always the data needed 
for school improvement efforts (Hough, Byun, & Mulfinger, 2018). Without resources to 
collect and analyze appropriate data, improvement efforts could falter. As one district 
administrator explained: 
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In terms of what's the gap between the theory and the practice... 
improvement science is harder than other educational initiatives to really 
pick up [even] once you have the ideas. And, | think there are many reasons 
for that. So, one is the difficulty of collecting the appropriate data. We often 
have data on the student outcomes. We don't often have data on the adult's 


practices. It takes time to do that [data 


collection]. It's especially difficult 


because these are person things so you need to either self-report or observe 


and those both have all these ch 


Districts’ responses to this need varied 
collection of observational data to iden 


allenges with them. 


. For one LIT, district staff supported the 


tify whether changes being tested resulted in an 


improvement in a specific metric measuring teaching. This district also freed a member of 
their research staff to collect and analyze qualitative exit slips designed to understand the 
socio-emotional aspects of students’ experiences in mathematics, easing the time burden 
for teachers when they worked with qualitative data. 


Coordinating up, down, and across the system is a potentially invaluable role of 


districts. By definition, classes are nested with 
other through broader district sys 


as a liaison to, as one DIT lead described, 


... facilitate the gathering of th 
setting their meetings, and 
district support, and organ 
staying in touch wi 
regularly about wo 


and making sure th 


izing or arra 
rking with 


at a lot of 


A liaison to disseminate information is 
struggle to be regularly involved in 


for cabinet-level representatives wh 


tems. Ata mi 


e school 
helping them make connections with their local 


th Our... advisor from 
the schools, what's happening at the schools, 
the other pieces are pretty well organized. 


in schools, which are connected to each 
nimum, it was important that the DIT served 


teams, and ensure that they are 


nging our follow-up meetings, and 


CORE, to communicate pretty 


particularly important when key personnel 


the ongoing work, which seemed particularly common 
o were stretched thin with multiple leadership roles. 


The coordination role, however, typically went well beyond the management 


and top-down communication functions des 
strategically selected the members of their D 
district structures 
created spots for 
DIT included the assistant superintendent wh 
district mathemati 
evel representatives. This mix of individuals h 
DIT directly to the cabinet, engage principals 
data support for the LIT, and reach into the c 


down to the schools with LITs and, often, in 
four of the district's school supervisor positions on the DIT. Another 


cs coordinator, a member of the district's re 


cribed by the DIT lead above. Districts 
ITs to build connections from the overarching 
to classrooms. One district 


the schools with LITs, the 
search staff, and cabinet- 
ad the potential to elevate things from the 
through their supervisory structure, access 
assrooms of the teachers participating in the 


oO oversaw all of 


LIT through district mathematics coaches. W 


ith this structure, the research lead learned 
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that the teachers on the LIT had conducted empathy interviews but lacked experience 
analyzing qualitative data. Sne was able to push in needed supports—taking their data, 
grouping student responses into categories, and then meeting with them to not only 
present the findings but also to show them her approach to analyzing qualitative data so 
that they could build those skills for future use. Additionally, information about the types 
of challenges teachers were experiencing teaching mathematics and the type of change 
ideas they were testing had the potential to influence the district's overall approach to 
supporting mathematics instruction. The DIT, therefore, became a potential conduit 
from classroom-level mathematics teaching up to the mathematics lead, providing her 
with a teacher-level perspective of the challenges teachers were encountering using the 
curricular materials and teaching approaches adopted in the district. 


The cross-level coordination role is also important because different levels of 
the system have access to different knowledge and resources. One of CORE’s coaches 
described a meeting she sat in on with a DIT, where a district leader with knowledge 
about diverse initiatives in the system was able to direct an unrelated resource to a gap in 
knowledge raised at the DIT meeting: 


| think that there's certain things that the district level people can see that 
school sites can’t in terms of opportunities. So, my favorite example that I’ve 
observed this so far is | was sitting at a [DIT] meeting where they were talking 
about [a continuous improvement aim around teacher retention], and they 
were talking about things that they were curious about in terms of people 
leaving [the focal schools]. And then one of the data people mentioned, 
“Well, you know we have that partnership with [a local university] where they 
conduct exit interviews for us. | Know that’s an unrelated project, but what 

if we funneled the [teachers from focal schools] through that [university] 
project so that we'd get [all teachers who were planning to leave] exit 
interviews?"... | said, “That's a fabulous idea.” That's an ongoing thing that you 
already have resources for. 


As the past two examples illustrate, all levels have unique insights into problems 
and potential resources. It is only when information is flowing freely up and down levels 
and laterally across different departments that the potential of the system can be unlocked 
(Burt, 2004). 


One note of caution is that it could seem enticing to have large DIT teams with the 
goal of making sure that a wide variety of voices are in the room. But this too comes at a 
cost. Several DITs struggled with getting consistent attendance at their meetings across a 
relatively large team. One DIT lead reflected on this problem, stating, 


| think one of the DIT learnings is that the membership at each meeting 
needs to be dependent on the problem, so it doesn't always have to be the 
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same people all the time. But we've gotta be clear about what it is that we're 
trying to do to have the right people in the room. 


The idea of having a small core team as well as a larger group of people who 
are asked to Join the meeting when the topic is particularly relevant to their role is one 
possible way to resolve this dilemma. 


Districts can also conduct their own improvement cycles, even around aims 
that are not solely situated at the school-level. An aim around improving mathematics 
achievement, constructed as it has been by the CORE districts, is a great example from 
which to illustrate a multi-level improvement journey. Clearly, there are classroom-level 
dimensions around instruction, but these might surface particular needs for teacher 
professional development and different instructional materials. For example, in one LIT, 
focused on trying to increase the amount of student-to-student dialogue in mathematics, 
the mathematics coach on the team learned from the PDSA that richer mathematical 
tasks were key to supporting more student discussion in mathematics. This district had 
spent several years working on a grant designed to support this type of instruction, yet 
teachers were typically not using the types of tasks that grant had tried to promote. This 
could spur the district to undertake its own investigation into the gap between desired and 
actual use of richer mathematical tasks in instruction and how they could redesign district- 
level Supports around use of those types of instructional resources. In 2017-18, we did not 
see many examples of improvement work at the classroom level triggering the next logical 
improvement work at the school or district level. 


In 2017-18, one district did run district-level improvement cycles as part of their 
CIC work. This district's broad theory of action was that to get to improving student 
outcomes for African American and Latinx students in Grades 4—8 they needed to start 
by addressing the central problem of having high turnover among teachers in some of 
their most diverse and low-performing schools. Unlike other CORE districts, their root 
cause analysis suggested that improving the instructional skills of the teachers currently 
in those schools would not lead to long-term gains in academic performance in those 
schools because teacher retention was too low. As a result, this district collected data 
about the problem of teacher retention in those schools and tested a few change ideas 
involving district and school administrators. The idea would be to move on to supporting 
instructional improvement within classrooms once the district has tested approaches 
to improve their retention of desirable teachers in these schools. This approach follows 
clearly from their root cause analysis in 2016-17 and provides an excellent example of 
how their change journey, unlike that of other districts in the CIC, might start at the district 


8 The district is pleased by the outcomes of this improvement work. Data show that, in 2017-18, there was a marked 
decline in avoidable turnover (i.e., turnover for reasons the school or district can control, which excludes things like 
teachers moving out of the region) in these schools, which is at least partially attributed to the work they did as part of 


the CIC. 
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level and then move down to the classroom level—and potentially back up at a later point 
in time—depending on the particular problem being addressed. This example serves as a 
reminder that for complex educational problems, there may be root causes in different 
aspects of the system, each of which could be involved in continuous improvement 
cycles coordinated over time by a DIT. 


Lesson 3: Context matters. Preexisting structures and processes, time for educator 
collaboration, and supportive leadership all influence continuous improvement efforts. 


In talking with experienced teachers, it is hard not to encounter a sense of reform 
fatigue (Payne, 2008). In recent years teachers have felt buffeted by No Child Left 
Behind, the Open Court “police,” pacing plans, Common Core State Standards, and new 
curriculum adoptions, just to name a few. Additionally, the new State System of Support 
being developed will, in all likelincod, provide supports around continuous improvement 
as the state’s new strategy for helping turn around its lowest performing schools and 
districts. One of the challenges for continuous improvement work, therefore, is to enter 
schools without being perceived as yet another “flavor of the day’ initiative or outside 
mandate with which to grudgingly comply. We would argue, however, that continuous 
improvement could be fundamentally different from preceding waves of reform. Instead 
of being a “new idea” about teaching, improvement science offers a systematic toolkit to 
make sense of new ideas and adapt them successfully for students, or to see gaps in the 
current set of operating procedures and innovate to address those gaps. 


Although the CIC's charge this year was focused on launching and supporting 
LITs, in reality, districts’ actions varied widely. No site was a vacuum. Extant programs, 
structures, processes, and relationships shaped how districts enacted the CIC work this 
year. This section outlines what we have learned about important interactions between 
what exists in districts and continuous improvement, based on the CIC’s work in 2017-18. 
We garnered three crucial, if not surprising, understandings that could help the state’s 
continuous improvement strategy succeed: 


1. When continuous improvement tools are used to address challenges schools 
already face, they can leverage current efforts and create a productive synergy. 


2. The introduction of new improvement science tools may initially create cognitive 
dissonance as participants construct how to best integrate their essential features 
with extant improvement processes and infrastructure. 


5. For school teams to successfully engage in continuous improvement, 
foundational conditions need to be met; for schools that struggle meeting those 
conditions, additional support from the district may be needed. 
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Continuous improvement is not a new approach to teaching and learning 
mathematics. Instead it is an approach that can be leveraged to address a wide array of 
gaps between current and desired performance. When schools perceived CIC work as 
an approach to existing work rather than as a separate initiative, we saw the work gain 
traction within participating teams. For example, one district selected schools to form 
LITs based on their work with a complex approach to mathematics instruction. While 
the approach has substantial research-based evidence of positive impacts on student 
outcomes when it is well implemented, it requires significant departures from traditional 
instruction and teachers may find it difficult to use successfully. The district has provided 
professional development on the approach for a few years but had not yet consistently 
seen the results on student outcomes that the research-based evidence indicated they 
should. In 2017-18, this district selected schools working on this mathematics approach 
to launch LITs, and teachers in one school dug in particularly deeply to CIC work. The 
work allowed these teachers to explore aspects of the mathematics program that they 
found particularly difficult to implement well, running PDSAs to hone their approaches 
to stimulating more discussion around mathematics. Although factors other than just the 
CIC are likely involved, initial reports of SBAC scores from this district show that students 
growth in mathematics performance at this school were substantial, with students in 
participating grades making above-average growth during 2017-18. 


Bringing CIC work around improvement into schools with existing structures and 
processes was more complex because it could create cognitive dissonance between 
preexisting and new improvement approaches as well as potentially providing knowledge 
and structures on which to build. To illustrate the challenges and benefits associated with 
integration of old and new improvement processes, consider an example from one district 
with improvement structures and processes that pre-date its work with the CIC. This 
district's professional learning communities used a framework called a Cycle of Inquiry to 
improve practice. Figure 5 shows a visual representation of the Cycle of Inquiry and the 
PDSA Cycle the CIC introduced. 


edpolicyinca.org &) 


VA PACE 


Figure 5. PDSA and Cycle of Inquiry 
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In many respects, the two improvement cycles map onto each other well: Both 
cycles prompt teams to decide on a question or objective, plan and execute a test 
or inquiry, and collect and analyze data from the test or inquiry. Yet there are notable 
differences. For example, the PDSA cycle emphasizes making a prediction and comparing 
data to the prediction, which is not a distinct feature of the Cycle of Inquiry. Research has 
shown that teachers’ existing Knowledge, belief systems, and context play a critical role 
in shaping their experiences with new ideas and approaches (Spillane, Reiser, & Reimer, 
2002). From this research we know that the similarity between the cycles could create 
confusion or dissonance for teachers and school leaders faced with adopting or adapting 
to improvement science approaches from the CIC (e.g., the PDSA cycle) into their existing 
structures and processes around improvement. Possible responses include rejecting the 
new ideas (which is unlikely given how schools and teams were selected—and we did not 
find any instances of this from LITs in 2017-18), attempting to adopt the new approaches 
wholesale (we also saw no examples of this in schools with existing improvement 
processes and structures), or attempting to adapt the new approaches (e.g., pulling 
specific tools into existing structures). 


Most typically, schools with existing continuous improvement approaches chose 
to adapt the improvement science approaches presented by the CIC. In one school, 
for example, a principal saw the PDSA cycle as it was presented in LIFT as potentially 
magnifying the strengths of processes or structures her school already had in place. She 
described her school’s data cycles, explaining that the leadership team compiled and 
analyzed detailed school data three times per year; additionally, teachers at the school 
had time to meet with their grade level teams and cross-grade content teams weekly for 
instructional planning. When introduced to the CIC's work, the principal realized that the 
improvement science tools offered more focus and direction for the extant data reviews 
and teacher meetings: 


[Our DIT lead] helped me understand the coherence between the teams 
and having this [improvement science] structure for doing the work.... So the 
whole systems part was made more visible to me.... The work [with the CIC] 
that I’m doing now is helping me understand it needs to be way more micro 
than what we were making it. 


Leaders who can thoughtfully hitch improvement science tools onto structures 
that already exist can capitalize on already-strong school habits and simultaneously bring 
new clarity to their purpose. However, if decisions about adaptations are made by people 
lacking a full understanding of the CIC’s approach to improvement science, they might 
accidentally distort some of the original intent of the tools or gloss over key differences. 
It is very difficult, in the abstract, to figure out when adaptations that increase the depth 
and speed of adoption will be beneficial over the long run or detrimental because they 
represent problematic deviations from a theoretical ideal. 
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Even while understanding the potential challenges involved when educators 
make sense of new information about improvement from the CIC in light of their 
preexisting knowledge and beliefs, it is also important to recognize the benefits of existing 
improvement structures. In the case of the school described above, where teachers 
were already familiar with the Cycle of Inquiry, time set aside for the Cycle of Inquiry was 
seamlessly dedicated to the CIC. Using existing teacher team structures solved some basic 
ogistical problems other sites struggled with, such as finding a common meeting time to 
work on team objectives. Professional learning communities also often offered conducive 
improvement structures for the CIC. 


In moving towards continuous improvement as a widespread way of working, 
there is the impulse to establish improvement procedures in the most struggling schools. 
However, severely struggling schools may lack the enabling conditions needed for 
continuous improvement to happen. For example, for PDSAs to be meaningful, teacher 
teams must be able to meet regularly. If they don't, the work may never gather the 
momentum necessary to improve performance. District leaders, therefore, should 
be prepared to bring in extra supports to allow improvement to happen in the most 
challenged schools. At one school, a teacher listed barriers to simply convening LIT 
meetings to which many teachers and principals may relate. As she explained: 


| will say unfortunately a lot of things happened. We had a [major school 
event], we had a teacher getting sick... we had a [schoolwide emergency] 
today.... Last Wednesday | was supposed to go to [an LIT meeting], and then 
it was canceled... and next week we have the break.... Originally | did [the 
LIT] with the other [teacher on special assignment], but | think she’s kind of 
not involved in this anymore, because of the change of leadership [that the 
district was undergoing... | have become the only one actively involved in 
this CORE improvement, because the principal is too busy. 


As this educator described, emergencies, conflicting priorities, and staff turnover 
all threatened the work of this LIT. However, her district elementary math coordinator and 
hub coach both maintained close contact and provided support. The continuing support 
from the district allowed this LIT to undertake continued improvement work, albeit more 
slowly than might have occurred otherwise. Having the cross-level supports and the key 
members of the DIT acting as champions of the work, as we discussed in the section 
on district roles, allowed this LIT to continue Its improvement journey. Nonetheless, it 
serves as a reminder that continuous improvement ideas are unlikely to lead to success 
in schools where the context presents substantial challenges to regular and substantial 
teacher collaboration unless there is a deep district commitment to supporting those 
schools and overcoming the contextual barriers. This lesson is particularly important 
to keep in mind as districts across California start to use continuous improvement 
approaches to work with schools identified on the State Dashboard for improvement. 
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We turn to one final aspect of context that is critical for success in implementing 
continuous improvement—supportive leadership. At the center of improvement 
science Is a trust that practitioners—teachers—not only can become leaders of their 
own improvement journeys but that they must become engaged as leaders for true 
improvement to take place. For continuous improvement to fulfill its promise, therefore, 
positional leaders (e.g., principals, district administrators) need to give teachers space to 
try, fail, process, and retry. PDSAs are about testing possible solutions, including ones that 
do not initially work, with failure sometimes providing the biggest learning opportunities 
because they can force questions about deeply held assumptions. Allowing teachers 
the space to work in this way may require a change in leadership culture, especially in 
struggling schools used to receiving mandates and oversight. Harris (2011), describing 
more broadly the shifts required of principals to change to a distributed leadership 
approach, writes the following: 


What distributed leadership means for principals is a fundamental change in 
their understanding of leadership and in the ways they enact their leadership 
roles. It implies the relinquishing of some authority and power, which is not 
an easy task, and a repositioning of the role from exclusive leadership to a 
form of leadership that is more concerned with brokering, facilitating and 
supporting others in leading innovation and change. It will require a different 
conception of the organisation, one that moves away from the bureaucratic 
to the collaborative. It will also mean the development of new skills and a 
new repertoire of approaches that fit the new role. (p. 8) 


Part of the challenge around creating a support context for continuous 
improvement, then, is finding and nurturing leaders in the most struggling schools who 
have a particular approach to the principalship that can empower cadres of teacher 
leaders to drive organizational and instructional change. 


Understanding how to make improvement work happen in an array of districts that 
vary so widely is a challenge to the CIC’s working together. A hub member put it this way: 


[CORE is] across the whole state, so things could look so vastly different, and 
they do look different in [one district than they do in another], which is okay. 
But we have something to anchor to what is different and why is it different 
versus everyone doing their own thing. 


As the language and toolkits of continuous improvement spread across the state, 
it will be increasingly critical to understand how improvement work will interact with 
what districts and schools are already doing and experiencing. Examining what factors 
exist and how they could affect improvement work is an important step in establishing 
high-leverage, effective teams. It will also be important for researchers, practitioners, and 
policymakers to keep exploring and discussing what continuous improvement looks like 
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in different contexts so that as each district or school adapts it in their own contexts, the 
essential components of what it takes to achieve sustained improvement will be in place. 


Lesson 4: District and school leaders are excited about the potential of continuous 
improvement to spur deep and lasting improvement. 


We close the lessons learned with a note about the promise of continuous 
improvement. Often educational reforms feel like something that is done to educators; 
their responses, not surprisingly, are to make personal decisions about how much they will 
invest in the implementation of the reforms. Common refrains among veteran educators 
about reforms are “this too shall pass” as well as a sense that whatever is being pitched as 
"new" is actually something old repackaged. 


In 2017-18, the CIC worked with eight DITs and supported the launch of about 
20 LITs. The local teams, especially, were carefully selected—first when districts picked 
schools strategically and then, within schools, as leaders determined which team of 
teachers should pilot the work for their school. By design, this set of teachers and schools 
was handpicked for the work and the selection process did sometimes include leaders’ 
beliefs that the schools and individuals involved were more likely than some to get excited 
about the work. Even with that caveat, we consistently heard reports from people at 
all levels of the system—district leaders, school leaders, and teachers—that continuous 
improvement felt different than most typical change efforts. As one teacher put it, 


My first, initial introduction with it was, it was different than any other 
approach that I've ever experienced with working in this district. And it was 
kind of refreshing to see that it was coming at improving our practices from 
a scientific place of looking at what's going on and identifying a problem, 
taking some baseline data, and then coming up with some ways to try to 
solve the problem, rather than, “This is a problem, it needs to be fixed, you 
figure out how to fix it.” Which is Kinda been sometimes how things come 
around. It’s like not really knowing what the problem is, and then trying to 

fix it without really knowing what it is, or assuming that It’s one thing without 
checking to see if it's that, and going at it in a strange direction. 


This teacher's explanation highlighted how the systematic approach, especially 
the early stages of collecting data to identify the root causes of problems, was powerfully 
different than her prior experiences with reform. A principal at a different school appreciated 
the promise that continuous improvement holds for learning from initial failures: 


In my opinion what's different is... going back to the PDSA that we use, 

and we try something and instead of just saying, “Well this didn't work, I’m 
dumping it.” We're now looking at it as, “Well this didn’t work, what must be 
tweaked? What must be fixed in order for It to work?" That's different. Some 
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of us are So ingrained into, “Well this was garbage. | just dedicated a whole 
week of planning for this but now it’s trash,” instead of, “Let me dedicate a 
little bit more time and see what didn't work, let me dissect into that a little 
bit more and see what didn't work to make it work.” 


These statements both suggest that the educators engaged in improvement work 
see it as a way of breaking the counterproductive cycles of introducing new initiatives 
only to discard them a couple of years later when they failed to achieve desired results. 
By making sure a problem is correctly identified, finding a solution that matches the local 
problem, and then giving educators a toolkit with which to improve implementation, 
continuous improvement has the potential to slow the constant churn of reform. 


Evidence from this year of implementation suggests that the energy and the will 
built up through LIT work may be critical to the long-term success of the improvement 
efforts. For example, one LIT lost two members who went on leave for unrelated reasons 
during the year. With most of the team gone, it was unclear if the team would carry on. 
However, one of the remaining LIT members continued to champion the improvement 
work, attending LIFT and continuing to be an advocate. Instead of dying out, the CIC work 
was amplified: The principal became interested in involving the whole math department in 
improvement work in the coming school year. 


That type of soread—the spread of work to more teams within schools and then 
to more schools within the districts—may be a critical marker of future success. Because 
while engaged educators see the potential in this approach, continuous improvement 
requires a steady investment of time for learning new skills, developing new cultural norms, 
and practicing improvement. As a result, there is already a pressure for the new approach 
to deliver results in spite of the fact that LITs just launched in 2017-18 and, in most cases, 
represented just one team of teachers in each school. As one DIT lead explained, 


For all the investment that we're making in this, what is the bigger outcome? 
Right? Because, yes getting [local improvement] math teams at [a few pilot] 
schools is good work. But, that’s not going to be compelling enough to 
participate in a relatively expensive collaboration that takes a lot of time 

and capacity, and so the payback has to be bigger. And I’m not saying it has 
to be immediately taken to scale. To me all along, | was kind of thinking, 
"Hey, [a few] local improvement teams, that’s not enough. We need work to 
influence our entire system to really justify the investment.” 


Similar sentiments were expressed by leaders in multiple districts. The implication Is 
that for continuous improvement to achieve the potential that its early adopters in CORE 
districts believe it could have, the energy needs to spread even while initial teams build 
their skills to a point where they can demonstrate evidence of the impact that this work 
has had for their students. 
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Conclusion 


In 2017-18, the CORE Improvement Community (CIC) took the next step in 
operationalizing what it could look like for eight large districts, serving approximately one 
million students, to collectively build their capacity around continuous improvement. 
The CIC added direct support for school- and teacher-level work so that classroom- 
level drivers of mathematics achievement (e.g., instruction, students’ feeling of self- 
efficacy in mathematics) could be addressed through CIC work. The CIC continued to 
provide multiple entry points for district leaders to learn about and apply improvement 
science approaches. In the process, the CIC took its next step in operationalizing what 
improvement science can look like at scale in education. This report described four 
main lessons that we think others should consider when thinking about continuous 
improvement work in other contexts. We conclude the report with questions that surfaced 
for us aS we considered the CIC's journey in 2017-18 and look towards the future within 
the CIC and other organizations seeking to support continuous improvement. 


The CIC's work in 2017-18 highlighted the complexity of implementing continuous 
improvement, with implications for districts that are trying to create the conditions that 
enable improvement to take hold. Looking forward, we think it is important for districts 
trying to do improvement work to consider: 1) How can districts create norms and 
processes to support Communication within and across organizational levels? 2) How 
can districts support leaders to develop the types of leadership roles and approaches 
necessary to create conditions for success? Addressing questions about fundamental 
enabling conditions and how they can be created will be critical for the success and 
spread of continuous improvement in education. 


Additionally, we think that the fact that the CIC has a mathematics achievement 
aim and a continuous improvement capability goal merits careful attention, because the 
ways the CIC navigates the resulting promises and challenges could be instructive for 
other networks in a similar situation. To that end, we think it will be interesting to track 
two questions related to the duality of the improvement goal and mathematics aim as 
the CIC emerges: 1) Have districts and schools built their improvement capability by using 
improvement science tools to work on the CIC's stated mathematics aim? 2) How and to 
what extent has the CIC supported within- and across-district collaboration to accelerate 
progress on reducing inequities in mathematics achievement in Grades 4-8? Addressing 
these questions will help build the path forward for others seeking to implement the 
theory of improvement science in the current educational context. 


Finally, the excitement around the promise continuous improvement has for 
transforming educational institutions and their outcomes naturally creates pressure for 
those outcomes to be attained. Yet, the work of continuous improvement is deep and 
complex, so It would be unrealistic to expect results to improve drastically at scale over a 
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Short time period. But continuous improvement also requires an investment of resources, 
and it is, therefore, reasonable for leaders to expect evidence that their investments are 
bearing fruit. This leads us to ask: 1) What interim outcomes should organizations look 

at to Know if they are making expected progress? 2) How can leaders build a shared 
understanding with school boards and broader community stakeholders about what early 
stages of the improvement journey should look like? As our research team has interacted 
with many people across the state, we have consistently heard that success or failure 

in setting and communicating realistic expectations to various stakeholders—especially 
school boards, given their role in holding the purse strings—will be critical to ensuring that 
continuous improvement Is given an appropriate chance to work. In the coming year, we 
expect to make progress on understanding all of these questions for the work of the CIC. 
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